PiPNN 4/6: integrate disk build pipeline by SeliMeli · Pull Request #1291 · microsoft/DiskANN

weiyaoluo (SeliMeli) · 2026-07-29T11:53:23Z

Purpose

This PR connects PiPNN graph construction to the production disk-index builder.

It uses the existing DiskANN graph format. It does not add a PiPNN-specific index format.

Main changes

BuildAlgorithm accepts explicit PiPNN parameters.
The common builder validates PiPNN before expensive work.
Explicit PiPNN selection never falls back to Vamana.
The PiPNN adapter loads the dense matrix and calls diskann::graph::pipnn::build_graph.
The adapter selects the existing sampled medoid.
The canonical graph writer stores degree, start point, and adjacency rows.
The common pipeline continues PQ training and disk layout.

The PiPNN core returns rows for real points only. The disk pipeline still owns frozen points and persisted metadata.

Review order

Review build/configuration/build_algorithm.rs.
Review algorithm selection in builder/build.rs.
Review the PiPNN adapter in builder/build/pipnn.rs.
Review storage/bin.rs graph validation and serialization.
Confirm that PQ and disk assembly use the common pipeline.

Validation

Tests reject invalid PiPNN configuration before artifact creation.
Tests reject dataset shape and point-count mismatches.
Tests check degree, medoid, and frozen-count headers.
Tests confirm explicit PiPNN selection under a low memory budget.
Writer tests reject invalid degrees and IDs before file creation.
Windows GNU and AArch64 checks pass.

Stack

Stack 4/6. Depends on #1290. #1294 adds benchmark routes.

Copilot

Pull request overview

This PR integrates PiPNN graph construction into the DiskANN disk-index build pipeline and exposes it through the benchmark CLI inputs, including automated fixture-based tests to validate the new JSON configuration and execution paths.

Changes:

Add a BuildAlgorithm selector (Vamana vs PiPNN) with JSON-facing PiPNN parameters, plus memory-estimate-based fallback to Vamana for disk builds.
Implement PiPNN graph building adapters for disk builds and in-memory benchmark builds, including vector-store “flat prefix” access for dense layouts.
Add a shared cached CLI fixture runner and new PiPNN benchmark fixtures (disk + in-memory) to validate end-to-end CLI wiring.

Reviewed changes

Copilot reviewed 33 out of 37 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
diskann-providers/src/utils/rayon_util.rs	Exposes underlying Rayon pool reference for APIs that need `&rayon::ThreadPool`.
diskann-providers/src/storage/mod.rs	Re-exports new adjacency-graph save helper.
diskann-providers/src/storage/bin.rs	Adds `save_adjacency_graph` for canonical adjacency graph serialization.
diskann-providers/src/model/graph/provider/async_/fast_memory_vector_provider.rs	Exposes unsafe dense-prefix slice access to support PiPNN build path.
diskann-providers/src/model/graph/provider/async_/common.rs	Implements `AlignedMemoryVectorStore::flat_prefix` and adds a contiguous-prefix test.
diskann-disk/src/lib.rs	Re-exports PiPNN parameters (feature-gated) and exposes build-algorithm selection.
diskann-disk/src/build/mod.rs	Re-exports `BuildAlgorithm` and (feature-gated) PiPNN parameters.
diskann-disk/src/build/configuration/mod.rs	Introduces build-algorithm configuration module and re-exports its types.
diskann-disk/src/build/configuration/disk_index_build_parameter.rs	Adds `build_algorithm`, PiPNN constructor, PiPNN config conversion, and memory-estimate-based fallback.
diskann-disk/src/build/configuration/build_algorithm.rs	New: serde-driven `BuildAlgorithm` enum and PiPNN JSON parameter struct.
diskann-disk/src/build/builder/build/pipnn.rs	New: PiPNN graph build adapter producing a canonical on-disk adjacency graph.
diskann-disk/src/build/builder/build/pipnn/tests.rs	New: disk-build integration tests for PiPNN selection, fallback, and header correctness.
diskann-disk/src/build/builder/build.rs	Routes graph construction to PiPNN (when selected) while keeping existing Vamana pipeline as default.
diskann-disk/Cargo.toml	Adds optional `diskann-pipnn` dependency + feature, plus test deps for new tests.
diskann-benchmark/src/main.rs	Refactors registry creation and adds PiPNN fixture test hook (feature-gated).
diskann-benchmark/src/inputs/graph_index.rs	Adds PiPNN build-algorithm selection for graph-index build inputs (feature-gated).
diskann-benchmark/src/inputs/disk.rs	Adds `alpha`, makes `quantization_type` optional, and adds disk build-algorithm selection + validation.
diskann-benchmark/src/index/build.rs	Adds PiPNN in-memory build path leveraging dense flat-prefix when possible.
diskann-benchmark/src/index/benchmarks.rs	Wires PiPNN build path selection into benchmark build flow.
diskann-benchmark/src/disk_index/build.rs	Wires disk `BuildAlgorithm` into disk-index build parameter construction.
diskann-benchmark/Cargo.toml	Adds optional PiPNN dependency/feature and enables fixture runner for dev-tests.
diskann-benchmark/tests/pipnn/inmemory/stdout.txt	New: expected output for in-memory PiPNN CLI fixture.
diskann-benchmark/tests/pipnn/inmemory/stdin.txt	New: commands for in-memory PiPNN CLI fixture.
diskann-benchmark/tests/pipnn/inmemory/input.json	New: PiPNN in-memory input JSON fixture.
diskann-benchmark/tests/pipnn/disk/stdout.txt	New: expected output for disk PiPNN CLI fixture.
diskann-benchmark/tests/pipnn/disk/stdin.txt	New: commands for disk PiPNN CLI fixture.
diskann-benchmark/tests/pipnn/disk/input.json	New: PiPNN disk input JSON fixture.
diskann-benchmark-runner/src/lib.rs	Exposes new fixture runner module for tests/downstream crates (feature-gated).
diskann-benchmark-runner/src/fixture.rs	New: shared cached CLI fixture runner implementation.
diskann-benchmark-runner/src/fixture/tests.rs	New: unit tests for fixture runner behavior.
diskann-benchmark-runner/src/app.rs	Updates integration-test documentation and switches tests to the shared fixture runner.
diskann-benchmark-runner/Cargo.toml	Adds `test-fixtures` feature + optional tempfile dep for downstream fixture usage.
Cargo.lock	Adds/records `diskann-pipnn` and `serde_json` dependency changes.
.cargo/mutants.toml	Excludes new fixture overwrite behavior from mutation testing.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

diskann-disk/src/build/configuration/disk_index_build_parameter.rs:17

estimate_pipnn_peak_memory uses size_of::<...>() unqualified, but this module doesn’t import std::mem::size_of. This will fail to compile under the repo’s Rust toolchain.

use std::num::NonZeroUsize;

use diskann::ANNError;
#[cfg(feature = "pipnn")]
use diskann::ANNResult;
use thiserror::Error;

#[cfg(feature = "pipnn")]
use super::PiPNNParameters;
use super::{BuildAlgorithm, QuantizationType};

diskann-disk/src/build/configuration/disk_index_build_parameter.rs:113

DiskIndexBuildParameters is publicly re-exported (via diskann-disk’s public API) and this change removes Copy from its derives. That is a breaking API change for downstream users that relied on implicit copies; if this is intentional, it should be called out explicitly (and potentially gated or versioned accordingly).

/// Parameters specific for disk index construction.
#[derive(Clone, PartialEq, Debug)]
pub struct DiskIndexBuildParameters {

Copilot

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (3)

diskann-disk/src/build/configuration/disk_index_build_parameter.rs:204

disk_index_build_parameter.rs enables #![warn(missing_docs)], and the new pub(crate) fn pipnn_config has no doc comment. If CI promotes warnings to errors, this will fail builds; even if not, it adds new lint noise.

    #[cfg(feature = "pipnn")]
    pub(crate) fn pipnn_config(&self) -> Option<diskann_pipnn::PiPNNConfig> {
        match &self.build_algorithm {
            BuildAlgorithm::PiPNN(config) => Some(config.into()),
            BuildAlgorithm::Vamana => None,
        }
    }

diskann-disk/src/build/builder/build/pipnn.rs:49

pipnn::build_graph reads the full dataset into memory (read_bin) and then calls find_medoid_with_sampling, which performs another full pass over the dataset via VectorDataIterator (plus a sampled centroid pass). On large datasets this doubles I/O and parsing work even though the vectors are already in memory.

    let data =
        read_bin::<Data::VectorDataType>(&mut builder.storage_provider.open_reader(&data_path)?)?;
    let context = PiPNNBuildContext::new(
        config,
        &builder.index_configuration.config,

diskann-providers/src/storage/bin.rs:388

save_adjacency_graph is re-exported as a public API, but its signature exposes diskann::graph::AdjacencyList<u32>, forcing downstream callers to take a dependency on diskann graph types just to use diskann-providers serialization. If the goal is a narrow cross-crate adjacency entry point, consider changing this API to accept adjacency rows as plain slices (or any type that derefs/borrows to &[u32]) to avoid leaking diskann internals into diskann-providers’ public surface.

pub fn save_adjacency_graph<P>(
    adjacency: &[AdjacencyList<u32>],
    max_degree: u32,
    provider: &P,
    start_point: u32,

Copilot

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated no new comments.

Codecov Comments Bot (codecov-commenter) · 2026-07-30T08:45:09Z

Codecov Report

❌ Patch coverage is 97.13467% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.72%. Comparing base (175f0c8) to head (a766b97).

Files with missing lines	Patch %	Lines
...nn-disk/src/build/configuration/build_algorithm.rs	86.66%	6 Missing ⚠️
diskann-disk/src/build/builder/build/pipnn.rs	97.77%	4 Missing ⚠️

Additional details and impacted files

@@                  Coverage Diff                  @@
##           pipnn-stack/03-core    #1291    +/-   ##
=====================================================
  Coverage                91.72%   91.72%            
=====================================================
  Files                      572      574     +2     
  Lines                   114177   114524   +347     
=====================================================
+ Hits                    104728   105051   +323     
- Misses                    9449     9473    +24

Flag	Coverage Δ
miri	`91.72% <97.13%> (+<0.01%)`	⬆️
unittests	`91.67% <97.13%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
diskann-disk/src/build/builder/build.rs	`93.00% <100.00%> (+0.17%)`	⬆️
.../build/configuration/disk_index_build_parameter.rs	`97.72% <100.00%> (+0.45%)`	⬆️
diskann-providers/src/storage/bin.rs	`96.72% <100.00%> (+2.42%)`	⬆️
diskann-providers/src/utils/rayon_util.rs	`97.54% <100.00%> (+0.03%)`	⬆️
diskann-disk/src/build/builder/build/pipnn.rs	`97.77% <97.77%> (ø)`
...nn-disk/src/build/configuration/build_algorithm.rs	`86.66% <86.66%> (ø)`

... and 5 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.